Intrinsic and Extrinsic Evaluation of an Automatic User Disengagement Detector for an Uncertainty-Adaptive Spoken Dialogue System
نویسندگان
چکیده
We present a model for detecting user disengagement during spoken dialogue interactions. Intrinsic evaluation of our model (i.e., with respect to a gold standard) yields results on par with prior work. However, since our goal is immediate implementation in a system that already detects and adapts to user uncertainty, we go further than prior work and present an extrinsic evaluation of our model (i.e., with respect to the real-world task). Correlation analyses show crucially that our automatic disengagement labels correlate with system performance in the same way as the gold standard (manual) labels, while regression analyses show that detecting user disengagement adds value over and above detecting only user uncertainty when modeling performance. Our results suggest that automatically detecting and adapting to user disengagement has the potential to significantly improve performance even in the presence of noise, when compared with only adapting to one affective state or ignoring affect entirely.
منابع مشابه
Adapting to Multiple Affective States in Spoken Dialogue
We evaluate a wizard-of-oz spoken dialogue system that adapts to multiple user affective states in real-time: user disengagement and uncertainty. We compare this version with the prior version of our system, which only adapts to user uncertainty. Our analysis investigates how iteratively adding new affect adaptation to an existing affect-adaptive system impacts global and local performance. We ...
متن کاملEvaluating a Spoken Dialogue System that Detects and Adapts to User Affective States
We present an evaluation of a spoken dialogue system that detects and adapts to user disengagement and uncertainty in real-time. We compare this version of our system to a version that adapts to only user disengagement, and to a version that ignores user disengagement and uncertainty entirely. We find a significant increase in task success when comparing both affectadaptive versions of our syst...
متن کاملEvaluating Automatic Dialogue Strategy Adaptation for a Spoken Dialogue System
In this paper, we describe an empirical evaluation of an adaptive mixed initiative spoken dialogue system. We conducted two sets of experiments to evaluate the mixed initiative and automatic adaptation aspects of the system, and analyzed the resulting dialogues along three dimensions: performance factors, discourse features, and initiative distribution. Our results show that 1) both the mixed i...
متن کاملBenefits and challenges of real-time uncertainty detection and adaptation in a spoken dialogue computer tutor
We evaluate the performance of a spoken dialogue system that provides substantive dynamic responses to automatically detected user affective states. We then present a detailed system error analysis that reveals challenges for real-time affect detection and adaptation. This research is situated in the tutoring domain, where the user is a student and the spoken dialogue system is a tutor. Our ada...
متن کاملPredicting and Adapting to Poor Speech Recognition in a Spoken Dialogue System
Spoken dialogue system performance can vary widely for different users, as well for the same user during different dialogues. This paper presents the design and evaluation of an adaptive version of TOOT, a spoken dialogue system for retrieving online train schedules. Adaptive TOOT predicts whether a user is having speech recognition problems as a particular dialogue progresses, and automaticall...
متن کامل